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ABSTRACT 

An evaluation of the Mathematics and Computer 
Assisted Remedial Teaching (MASCART) project in the Pre-Entry Science 
Course (PESC) at the University of Botswana focused on two questions: 
(1) Considering the conditions for successful implementation 
highlighted in the literature, has the implementation of computers in 
the PESC department been such that institutionalization can be 
expected? and (2) Did the use of the MASCART software result in 
knowleage gains? A case study design was utilized for the first 
question and a pretest-posttest design for the second. Most of the 
factors inflaencihg the success of implementation were fulfilled. 
Although the institutionalization phase is not yet complete, 
developments are promising. Based on the case study design, it can be 
concluded that there is no reason to doubt, for the time being, the 
validity of the theoretical starting point of the study — that the 
implementation and institutionalization of an educational change 
depend on the existence of specific conditions at the level of the 
materials themselves, the institution, and the society. With respect 
to the second question, a general improvement in basic algebra 
knowledge and skills was measured. While this gain cannot be 
unequivocally attributed to the quality of the MASCART materials, 
data indicate that the combination of increased time on task and 
working with MASCART materials was effective. (16 references) 
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SUMMARY 



Jacqueline Piion 



This paper describes the evaluation of the Mathematics and Sdence Computer Assisted Remed»t Teaching (MASCART) 
project in the Pre-Entry Science Course (PESC) at the University of Botswana The evaluaaon focussed on the following 
questions: 

- Considering the conditions for successful implementation as mentioned in the literature (e.g. Fullan. 1982). has the 
implementation of computers in the PESC Department been such, that we may expect insfitutionaTization ? 

- Did the use of the MASCART software by students indeed result m knowledge gains? 

For the first question a case study design was applied, for the second one an experimental design. It coukf be concluded 

that most of the factors influendng the success of implementation were fulfilled, while some weak points couW 

detemiined. Although the phase of institutnnalizatnn is not yet completed, developments are promising. 

With respect to the second research questnn. a general improvement in basic algebra knowledge and skills could be 

measured. We couW not proof that this improvement was caused by the quality of the MASCART materials, tut the data 

show that the combination of increased time on task and the working with MASCART materials was effective. 

Based on the case study design, it couW be concluded that for the time being we have no reasons to doubt the validity of 

our theoretical starting points, that certain factors influence the implementatwn and institutwnalizatton of an educational 

change. 



1. INTRODUCTION 

This paper descritws the evaluation of the Mathematics and Science Computer Assisted 
Remedial Teaching (MASCART) project in the Pre-Entry Science Course (PESC) at the 
University of Botswana. Botswana has an educational system that is still developing towards a 
more definite shape. The secondary school (a typical product of the British leading role in the 
southern African region) tal<es five years and school leavers who wish to enter the Faculty of 
Science of the University of Botswana first has to take PESC . PESC is a remedial bridging 
course between secondary school and university, that lasts for about 7 months, covering topics 
from Mathematics, Chemistry, Physics, Biology and Language and Study skills. The training of 
basic conceptual ideas, cognitive and practical (laboratory) skills and methods in these subjects 
are considered to be more important than acquisition of pure facts. The number of students 
participating in the course is increasing every year since the start of the course in 1975. In 
January 1989, about 300 students entered the PESC course. When still in secondary school, 
students are selected for entering PESC by a PESC-lecturer team. This selection is based on 
students performance in the Can*ridge Overseas School Certificate (COSC) or on PESC's own 
selection tests, and on a recommendation from the secondary sch- ' The best students of a 
grade cohort are thus selected for following the PESC course. 

A report on the use of computers in the University of Botswana in 1985 was a starting point for 
considering the introduction of computers in the department of PESC; assistance was provided 
by the Basic Science Program Unit (BSPU) of the Free University Amsterdam (the Netherlands). 
A computer laboratory, equipped with 30 micrccomputers, has been available from 1989 onwards 
and a (compulsory) computer awareness course was introduced in the PESC curriculum. The 
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facHity of a computer lab also enabled the introduction of computer assisted instruction in 
Mathematics and Science, especially for remedial purposes. The provision of remedial 
mathematics and science teaching through the computer is an advantage in the PESO setting, 
because other methods of remediation (e.g. by the lecturer or by peer tutoring) are not possible. 
This would require extra manpower, which is not available. 

Students entering the PESO course come from different secondary schools, which causes great 
variation in knowledge level among students. Although all students passed their ordinary {'O') 
level exams, many do not really master the basic science and mathematical concepts. Thouoh 
PESO is remedial in itself, and the same concepts are taught in PESC as already were taught in 
secondary school (with small extensions), still many students have difficulties with the subject 
matter. Extra individual remedial teaching via computers was expected to be a very helpful tool in 
this situation. 

Smit and Wolff (1989) reported about the learning difficulties of PESC students in Mathematics 
(especially in Basic Algebra), caused by misconceptions which students have. Traditional 
teaching in PESC did improve the understanding of fundamental concepts in certain areas, but 
the formation of other concepts was still unsatisfactory. The suggestion made by Smit and Wolff 
that additional instruction with the computer may change this, resulted in the development of 
experimental software for Basic Algebra: the MASCART-Basic Algebra project. The MASCART- 
Basic Algebra project also serves as a pilot, to examine the possibilities of this type of software 
for PESC-Botswana, as well as for ether, similar settings within developing countries. If this pilot 
would prove to be successful, the project could be extended to other subjects within PESC 
(Bfology, Physics, Chemistry) and/or to other countries. 

The main aim of the MASCART-Basic Algebra materials Is providing remedial instruction aimed 
at improving the algebra knowledge and skills of students by overcoming misconceptions, which 
students might bring in from secondary school or from the regular PESC lessons. The software 
(M ASCART materials) Is meant to hep students in remediating particular topics v/rthin Basic 
Algebra, by providing diagnostic exercises and specific feedback, students can follow the 
MASCART course individually, without the help of the lecturer. The MASCART materials are 
being used complementary to the usual teaching program for Mathematics, and as such add two 
periods (2 x 45 minutes) per week extra to the teaching progranune. As the MASCART project is 
the first educational computer application within the PESC Department, careful attention is given 
to the implenfientatfon of it on departmental, as well as teacher level. 

The introductton of the computer in Botswana can be seen as an educational change. Such a 
change process can be conceived as consisting of three stages: initiation, implementation and 
institutionalization (Fullan, 1982). In the initiation stage, the decision of introducing computers in 
the PESw Department was taken. Implementatfon, being the process of putting the change into 
practice, is an important part of the process of educational change. Stable, routine 
implementation is one of the conditions for ultimate institutionalization of a change, that is for 
making the change an integral part of the functioning of the Department. 

The evaluation of MASCART was initiated after Its implementation was prepared and organized 
by the FUA and the PESC department. The evaluation focused on the following questions: 

- Considering the conditions for successful implementation as nwntioned in the literature (e g 
Fullan, 1982), has the implementation of computers in the PESC Department been such that 
we may expect institutionalization ? 

- Did the use of the MASCART sof*ware by students indeed result in knowledge gains? 

Although misconceptions research \s the base for the devetopnwnt of the MASCART materials 
no further discussion will take place on the results in this light. Detailed analysis of the results has 
been done by Smit and Wolf and their reports discuss whether the misconceptions are changed 
since the Introduction of MASCART (Smit, Oosterhout and Wolff, 1989a, 1989b) 
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In the next section, first a theoretical perspective and conceptual framework will be discussed. 
Then, the design and method for the study will be presented, while section four deals with the 
results. 



2. PERSPECTIVE AND CONCEPTUAL FRAMEWORK 



in this project, the evaluation strategy of Brinkerhoff, Brethower. Hlutchyi and NowakowskI 
(1983), consisting of seven steps, has been applied. In this strategy, the third step consists of the 
development of an evaluation plan, in which the procedures and sources for information collection 
are described. An educational evaluation plan needs an underlying framework, which is the basis 
for selecting Information about the forces that influence student's achievement. A helpful 
framework can be derived from the th;ee dimenstonal cube of Hammond (in Worthen & Sanders, 
1987). The structure of this model is based on three sets of variables: population, behaviour and' 
instruction (page 66). When dealing with the 'population*, the different audiences playing a role in 
the project, can be included in the structure. 'Behaviour' can be seen as the goals and objectives 
for Introducing MASCART; this category can be divided Into the cognitive, affective and 
psychomotor domain. 'Instruction', being the process which will be evaluated, is in this study 
operationalised as those variables playing an important role in the implementatton and 
institutionalization of the MASCART materials. These variables can be derived from theories on 
the implementation of educational change. These theories imply that change in practice occurs 
when certain elements (or factors) occur in combination.Fullan (1982), Fullan, Miles and 
Anderson (1988), Verspoor (1987) and Van den Akker, Keursten and Plomp (1989) distinguish 
several factors for the successful implementation of an innovation. These factors, as summarized 
by Janssen Reinen (1989), are given in Tablel. 



Insert Tablel about here 



For Implementation, three levels can be distinguished: the material level; the institutional level 
and the institution transcending level. At each level several factors play a role. At the material 
level, three factors are important. In order for implementation to succeed, implementors have to 
gain a clear understanding o\ what to do and what to change in order to put the innovation into 
practice. The problem of subjective f- ' objective meaning of the change (that is. the possible 
discrepancy between the meaning o. receivers and the designers) belongs to this factor The 
complexity o\ a change is dependent on the degree to which three dimensions are of relevance- 
the possible use of new or revised materials, the possible use of new teaching approaches and 
the possible alteration of beliefs (Fullan. 1982). Consensus about the change is important ' 
because the prospects for successful implementation are greater when those, expected to carry 
out a change, agree on the need, on the appropriateness of the innovation selected, and on the 
priority of the change relative to other tocal concerns. Many changes fail to get implemented 
because the teaming materials themselves are insufficiently developed, wfiich indicates the 
Importance of the quality o\ an innovation. Furthermore, the innovation should be practical It 
should address students and teacher needs, it shoukJ fit well with the teacher's situation and ^ 
should include concrete how-to-do-ht information. 

At the Institutional level, eight factors need to be taken into account. The commitment and 
actions of (e.g. central office) administrators are critical to the success of implementation efforts 
Implementation is more likely to happen when there is clear, consistent communicatton and 
pressure from the administratfon, both initially and during the implementation. The process of 
irnplementation and institutionalization means that the people organizing and facilitating a chanqe 
effort have to be able to put together a clear organization model and a set of procedures for 
achieving implementatton and continuation of the change. Professional development and 
assistance are important for both the dissemination and implementatton of educattonal 
implementations. Substantial changes in practtoe are unlikely to succeed without foltow-up In- 



service training and consultative assistance during implementation. The success of the 
implementation is also highly dependent on the creation of effective ways of getting information 
about the iirplementation progress (monitoring) and problems.** The crux of the matter is to get 
the right people talking together on a regular basis about implementation issues. Furthermore, 
effective implementation depends on the prwc/pa/ taking an active role in initiating and or 
responding to change efforts within the school. The principle must create a positive climate for 
change. The outcome of implementation is susceptible to the influences of changes in the general 
organizational and social context, therefore the factor environmental stability must also be 
considered. Mutual support among the people worthing with the innovation as well as good 
communication are important, because everyone can learn from each others activities in the 
process of implementation. The motivation to use the change and the attitude towards the 
innovation of both students (users) and teachers are important, because they determine whether 
the innovation is actually going to be used. Therefore the last two categories: user values and 
preferences avd motivation and attitudes of teachers are important as well. 

At the institution transcending level, three factors are important. Rrst of all, comrwnity 
support. Research indicates that implementation of most educational innovations proceeds 
without much community awareness and involvement. But when community members do take an 
active interest in the adoptton and the implementation of particular innovations, their support for 
or against a change is likely to be a major factor in tocal decision making and commitment. 
Environmental stability deals with the economica! and political background of the country in which 
the process of change takes place. National context covers the 'power* in a country, and, in this 
case, especially the policy of a Government towards the use of computers in education. 

For InstttutlonallzaUonxo occur, the same three levels as menttoned for implementation are 
important. At material level it is important that ttie change has proved its worth in the pitot phase 
of implementation. Otherwise, continuation of the change is not necessary. 

At the Institutional level, a stable and routinely irr^lementation means a process of pilot use of 
the Innovation without too much disturt)ances and with the implementation factors all taken into 
account. Inbuiltness of the project means that on the tong term the innovation must not be seen 
as new a ny more. A long term policy must be formulated, whteh takes as starting point regular, 
self-evklent and increased use of the innovation in the future. For computers, this policy couW 
consist of the planning of new investments, the anangement ol maintenance, etc. The factor of 
shared ownership and depersonalized patterns is closely related to the inbuiltness of the project. 
Every participant in an organization, and not only the eariy adopters, should have the idea that 
working with the innovatfon is possible. Therefore a good interaction among all people of an 
institution is extremely important. Ultimately, an active and motivated principal is not only 
important during implemsntatfon, but also during this phase. 

At institution transcending level, it must be clear that there is a continuing necessity of the 
change, which means that the change shouW contribute to the solutton of an educattonal problem 
on that level. Only when that is the case, support and planning\or instituttonalizatton is possible. 
This means that general policies conceming the introduction of the innovation need to be 
developed and that financial support from the government must be consWered. 

All factors are included in the evaluation model and will be used to evaluate and analyse the 
implementation and institutionalization of computer use in the PESO Department. Combining the 
sets of variables of audiences, goals/objectives and change factors in one model (Figure 1) 
provides us with a conceptual framework for tiie evaluation. 
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Meta-analyses of educational computer use (e.g. Hasselbring, 1986) show that: 

- the use of CAI improves student attitudes towards computer use in the learning situation; 

- the positive effect on learning achievement occurs regardless of the type of CAI used, the type 
of computer system used or the age range of students; 

- tutorial and drill modes seem to be more effective for !ow ability students than for middle or high 
ability students. 

These results are based on research situated in more developed countries. This study 
investigates whether the same type of results also have validity in a less developed country. 

Based on the above theoretica* principles, the two general research questions have been 
operationalised into more concrete evaluation questions: 

1 .1 Are enough resources (budget, time, personnel) available for the total MASCART project and 
are sufficient preparations taken (on meso- as well as on micro level)? 

1.2 Is the project sufficiently supported in the PESC department (on meso- as well as on micro 
level)? 

1.3 Is the communication between relevant audiences adequate? 

1 .4 Are there any factors in the environment which influence process of implementation in a 
negative way? 

1.5 Is the organisation (help available, other materials, problems) during the MASCART sessions 
adequate? 

1.6 Are students positive about the new technology (compute^ and the MASCART materials? 

1.7 Is the MASCART software of good quality (considering presentation, instruction and 
mathematical content)? 

2.1 Are there general gains in Basic Algebra knowledge for PESC students usir>g MASCART. 
when compared to students who do not? Stated in other words: dkJ the materials indeed prcof 
their effect? 

2.2 For which ability group of students (low. middle or high) is MASCART more effective? 



3. RESEARCH DESIGN 



3.1 Evaluation of the Impr ementatfon of the MASCART materiars 

A case study design, using different instruments (observations, togbook- and program analysis, 
questionnaires and interviews), is the best research design to study the implementatton of 
MASCART. the computer assisted remedial teaching (Yin.1988). The case study will show 
whether and why the implementation process has been successful or not. The assumptton is that 
one can expect successful implementation, if during the implementatton process the theories for 
implementing change (as operattonalised in the preceding sectfon) have been foltowed. If the 
case study indeed confirms this assumption, this study then replicates the conclustons of 
Verspoor (1987). and contributes in this way to the valWity of theories on educattonal change in 
the context of less devetoped countries. Yin (1988) speaks in this cor:text of replfcatton togfc. 
However, if implementatton of MASCART will not be successful, then our theoretfcal assumptton 
is apparently not the right one. This whtoh will force us to refonnulate the limits of valWity. 

Information sources 

To ancwer the questtons concerning the success of the implementatton. data were collected by 
means of interviews, questtonnaires and observatton from Mathematfcs lecturers, other PESC 
staff, software developers. Basto Science Program Unit of the Free University, the cormuter lab 
assistant and the students. 

The interviews and the questtonnaire are devetoped on base of the evaluatton questtons. I'a Girder 
to collect the relevant informatton from students, while using computers, an observation 
iristrnment is used. The observatton scheme was developed based on the results of an eariier 
pilot test of the MASCART materials, the classroom observatton scheme of Hanron (1980) and a 
first week of non-systematto observatton of a;i students foltowing MASCART; the observatton 
scheme was pitoted m the second week. The main observatton categories are: 



- starting procedure; 

- during the program: asking help, use of scrap paper, guessing, disturbances; 

- ending procedure. 

Furthermore, the computer lab a-^istant kept a logbook in which information was recorded, like 
assistance given to the students, technical problems that arose, influencing environmental factors 
etc. This togbook has been developed based on the results of the first week's observation of the 
assistant. 

3.2 Evaluation of the effects of MASCART 

in order to investigate the effects of MASCART, an experimental design has been developed. 
The following consideralion had to be taken into account. The establishnrent of a control group of 
students, which would not receive any remediation at all, was morally not justifiable. This control 
group would be deprived of the chance of possibly getting better results, which they really need 
for getting a good recommendation after PESC. It was therefore decided that the control group 
should be able to work with MASCART after the experimental period, but at a time which was 
before the PESC examinatfon. 

Test design 

A test-design has been set up, using the notation of Campbell and Stanby (1963). This desian is 
depicted in Figure 2. 



Insert Rgure 2 about here 



The total group of 260 PESC students (40 students left during the course) has been pre-tested in 
February 1989 after PESC-unit 1 (01), using a 40-item multiple choice test on Basic Algebra (KR 
= 0.80). A nomial PESC unit consists of lectures, practlcals, tutorials, assignments and weekly 
achievement tests. Based on the results of this pre-test, the students have been matched (M) Into 
equivalent experimental and control groups. The matching took place within the different PESC 
groups in order to keep other conditions (like kind of lectures, practfcals and tutorials) as Identical 
as possible. During Unii 2 of PESC, the experimental group received extra remedial teaching, 
using the MASCART materials that cover unit 1 (treatment X). This meant spending an extra two 
periods (2 x 45 minutes) per week on Basic Algebra. The control group only received the regular 
teaching of Unit 2 consisting of 8 periods of lectures, practicals and tutorials. A tutorial means 
time for students to ask questfons about topics discussed that week. After Unit 2, the same Basic 
Algebra test (using a different sequence of questions) has been administered as post-test (02, 
KR = 0.80). Afterwards, the second group received the treatment X. It is possible to measure the 
overall effect of cognitive knowledge gains by comparing the test results of both experimental 
group and control group on pre- and post-test (01 and 02). Concerning the differential 
knowledge gains, two expectations exist: 

- findings of e.g. Hasselbring (1986) indicate more effect of CAI for low ability students than for 
middle of high ability students; 

- PESC lecturers, based on their teaching experience, expect that the middle ability group would 
gain most from MASCART. ^ *^ 

To test these expectatfons, the total group of students has been divided into different ability 
groups. This has been done in two different ways, which will be explained later in 4.2.2 and 4.2.3. 

Instruments 

The Basic Algebra Test (BAT) used in this study, is not representative for the complete PESC 
content, but was developed to cover those parts with which students have learning difficulties 
(according to the experience of the lecturers). Pre-piloting of the test took place in June 1963 
The coefficient alpha - all items proved to be .74, the Spearman-Brown coefficient was 72 
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4. RESULTS 



In this section, the results of the research concerning the two questions will bs presented. First 
the process of implementation will be considered; then the knowledge gains of students will be 
handled. The answers to both questions provide the necessary information to express some 
expectations concerning the further institutionalization of the MASCART project in the future 



4.1 The Implementation process 

The factors that may influence the process of implementatfon, as stated in Table 1 will be 
discussed below. Each of the factors will be given a score, in order to visualise the success of this 
factor. A (-) means that this factor is not taken into account during the process of implementation 
while a (+) indicates that the factor Is fulfilled. If no such judgement can be given, or If the factor is 
not applicable, the score (0) Is given to the factor. 

4.4.1 Characteristics of the materials 

Considering the characteristics of the materials, that aro important during the process of 
implementation, four different aspects can be distingu^hed. 

Clarity and complexity (+) 

In this project, devetopers, PESC Department, project supervisor and teachers appear to be well 
informed and know what Is expected from them In the project. In the PESC setting, the innovation 
is complex to some extent because it concerns the Introduction of an (almost) unknown 
technotogy. At the other hand, the Innovatfon does not require a re-arrangement of the total 
organization of PESC or the subjects, in particular because MASCART Is used In a remedial 
mode, extra and above the normal PESC classes. MASCART asks some extra time from the time 
schedule but the other subjects are not really affected by the change. Furthermore, the overall 
instruction strategies of the lecturers do not require any changes. This makes the change as a not 
too complex one and therefore easier to accept. 

Consensus/conflict about the change (0) 

From cariler project documents one cannot determine whether there is a real need within PESC 
for the Introduction of a remedial, computer assisted course. Project staff and representatives of 
the University of Botswana and the Free University do agree that the MASCART project is 
appropriate in this setting. 

Quality (and practicality) of the change (+) 

External and internal sources judged that the mathematical content of MASCART is generally 
good (although minor revisions still can be made). The Instructions In the program are dear and 
the use of a limited number of keys Is consequently dealt with. The program Is mainly computer 
controlled. This was considered to be the best option for this specific target group, which Is not 
used to control their own learning process. The feedback Is designed very carefully and reacts on 
the specific mistake made by the student. The presentation style Is used consequently The 
question whether the MASCART materials Indeed correct misconceptions is answered bv Smit 
Oostertiout and Wolff (1 989a, 1 989b). 

The opinfors of the students about the quality of the software are collected through a 
questionnaire. Most studems very much like to wori< with MASCART. In their opinion, the 
instructions are very clear and the feedback was considered to be very useful. Observations 
during the MASCART session showed that the students worked with much concentration. In 
summary, f! can be concluded that the MASCART materials are of good quality. 

Relevance (+) 

The PESC lecturers do not have extra time needed to assist Indivklual students with their 

P'^oielf • I?'^ ^ *° ^'^^"9® "83^ ^^re because the number of students 

in PESC will keep growing in the coming years. Although MASCART was not Initiated from a 
perceived need of the PESC Department, the staff judged it as highly relevant 
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4.1.2 institutional level 

On Institutional level, several factors are Important to take into account during the process of 
Implementation (Table 1). The Implementation of MASCART was not organized on the basis of 
these factors. Afterwards, the goodness of fit of these factors was reconstmcted. 

Central office direction, commitment end support (+) 

For this factor two institutions need to be taken into account in this project, the Free University 
Amsterdam and the Department at the University of Botswana. Money and materials came for a 
great deal from the Free University (and international donor organisations). This university 
provided a great deal of expertise through the development of the courseware and the provision 
of support, where necessary. Time, operationalized in concrete implementation activities, was 
contributed by the Mathematics and Computer section of PESC, in which the Head of Department 
is a leading figure (see a separate factor below). 

Professional devetopment assistance and participation (0/-) 

Because MASCART is supposed to be lecturer independent, there is no immediate necessity to 
tram PESC lecturers for this specific innovation. Yet, it would be wise to have several people Jn 
the PESC Department who would have some technical l<nowledge of computers and who are 
familiar with TAIGA, the authoring system used to develop MASCART. In the long term, it is not 
wise to rely for technical problems completely on the experience of one computer expert within 
the Mathematics section or on a computer company,. Also, the courseware development 
expertise of the Free University must not be the c V source for educational software. 

Implementation monitoring and problem solving (•) 

In order to monitor the implementation, there should be a detailed plan of implementation and 
someone should be available to monitor the process and to solve problems during the process of 
implementation. In the PESC setting, this factor is not fulfilled. During this pilot phase, monitoring 
tool< place via the two available researchers. Problem solving Is, next to the limited help of the 
computer company, done by one lecturer who has computer experience. He has to do this next to 
his nonnal teaching job. If the computer lab assistant would be better qualified, she c«uld do this 
job. In the case of PESC, the assistant was not trained to solve technical problems arid, not being 
a Mathematician, she could not assist students in difficult mathematical problenis either. 

Principal's leadership (+) 

As already stated above, the Head of Department is a leading figure in the project. He is very 
enthusiastic about the project and recognizes also other possibilities of introducing the new 
technology in the department. Although not directly involved In the inplementation of the 
materials, he wants to be kept infonned about the activities within the project. 

Environmental stability (+) 

The stability of the environment can be considered in two different ways. The PESC Department 
as the environment of MASCART, is a stable, motivating environment for the introduction of an 
innovation. On the other hand we have the computer lab, as the physical environnwnt of the 
computers. When woridng with this technology (especially in the context of a less developed 
country), it Is important to place the computers in the right environment. The newly built computer 
lab Is dust free (dust covers are available for every computer) and air-conditioned and can 
therefore considered to be an excellent environment for the computers. The only problem in the 
beginning was that the air-conditioners did not wori<; however t ie high temperature did not have a 
noticeable effect on the computers. The other disturbing factor of the environment in this pitot 
phase, noise, is also considered to be a temporary factor with no disturt)ing influence in the long 



Networking (+) 

Communication among the different participants took place in a very infomial way. Within the 
project itself, communicatfon took place on an equality base of both counterparts (Free University 
Amsterdam and University of Botswana) in the project, because there Is a regular 
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correspondence between software developers and the computer expert of PESC. The mutual 
support among the colleagues of the department is good. However, Ijecause only one person 
could be considered as 'computer expert', he aione was d'^ing most of the activities. 

User values and preferences (+) 

From the observations, it could be concluded that students were very eager to learn and are 
highly enthusiastic. The student questionnaire also pointed at favourable opinions of students 
about the use of the MASCART software. 

Motivation and attitudes of teachers (0/+) 

Because the MASCART materials are meant to be lecturer independent and the lecturers were 
not directly involved in the execution of the experinent, this factor was not of very much 
importance during this first experience with computers in PESC courses. However, as already 
mentioned, staff and representatives are very nrotivated and have a positive attitude towards the 
project. 

4.1.3 Institution transcending level 

The factors mentioned in Tablel as being important on this level (community support, 
environmental stability and national context) are not applicable in this case because of the 
specific position of the project. The project started through cooperation between PESC and the 
Free University Amsterdam and is therefore relatively independent from national policy. Aims and 
values of the government do not have an influence on the project, although it is interesting io see 
that the project fits within the plans of the government to introduce computers in the educational 
system. Politicians in Botswana are nowadays very positive about the use of computers in 
education as this contributes to a better job qualification. At present, there is a great demand for 
people who have experience with computers. 



4.1.4 Synthesis 



The presented results of the process of implementation make clear that in general the process of 
implementation has been canled out without many disturt)ing factors. Moct factors have indeed 
been fulfilled, but concerning a few factors, improvements are possible. The conclusion can be 
that the implementation of MASCART within the PESC setting has indeed been rather successful. 



4.2 Knowledge gains 



4.2.1 General knowledge gains 

The hypothesis whether students using the MASCART materials (experimental group) perform 
better on the Basif Algebra Test (post-test), than students who did not uso the materials (control 
group), was tested one-sided ( « 0.05, Table 2). 



Insert Table 2 about here 



These results indicate that students in the experimf-ntal group indeed performed significantly 
better. Their Itnowledge and sltills in basic algebra have indeed improved more than Itnowledge 
and sidlls from students in the control group. Whether this improvement is caused by the quality 
of the MASCART materials or by the increased time on tasit (students of the experimental group 
receive about 2 x 45 minutes of remedial teaching per weel( extra) cannot be detennined from 
I.!lf/f 1^ "'^'^ dilemma will be discussed in more detail in ?<Krtion 4.2.3. On the other hand, if 
MASCART would not exist, no other fomi of remediation could have been possible because 
lecturers do not have time for this Idnd of activities. 
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4.2.2 Eff Ktt on ability group 

The sea- id question is whether the MASCART materials have different effects for the different 
ability groups (low, middle and high). This question has been examined by using two different 
methods. The second method will be explained in section 4.2.3. For the first method, the total 
group of students are divided into four quartiles; the low ability group consisting of the 25% with 
the lowest score on the pre-test, the middle ability group of the next 50% of the students and the 
high ability group of the top 25%. The MASCART materials have been developed for the low- and 
middle ability group especially, but the lecturers of PESC believe that the main effects will be 
found in the middle ability group. They expected that the entrance level of the materials may be a 
little too high for the low achievers. The null hypothesis to be tested is that all three ability qroups 
will equally gain from MASCART. 

Rgure 3 shows the mean scores on the pre- and post test for both experimental- and control 
group at the three ability levels. The two lines In the middle of the figure represent the results for 
both groups on the pre-test. The two lines are about the same which was to be expected because 
the matching procedure was based on the results of this test. The upper two lines represent the 
results on the post-test. The general effect of MASCART is shown, since the scores of the 
experimental group are higher. It can be seen from the results of the control gioup that the 
regular instruction in PESC (about other topics than Basic Algebra) also improve the scores on 
the post test. 



insert Figure 3 about here 



A regression analysis has been conducted to check whether the factor ability has indeed a 
significant effect. The results of this regression analysis are shown In Table 3. This Tableagain 
shows that there is indeed an effect caused by the experimental group. Also, the factor ability 
plays a role of significance. From these data it sannot be concluded which ability group benefited 
the most. Therefore, a nrwre thorough analysis iS ne&ded. 



Insert Table 3 about here 



Table 4 shows the differential scores (post-test minus pre- test scores) for the experimental- and 
the control group of the three ability groups. The two middle regression lines in Figure 4 are 
oased on these results. They indicate lat the low ability group Is benefiting relatively more than 
the middle or high ability group. The lowest line depicts the difference on the post test between 
the experimental and the control group. This line can be considered to be the 'netto effect* of 
MASCART. 



Insert Table 4 about here 



Insert Figure 4 about here 
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4.2.3 B«ner results due to more time on task or to quality of IMASCART? 



Next to tlie regular eight periods of instruction, students in the experimental group received two 
extra periods of remedial instruction per weel<, by working through the MASCART materials. The 
analysis applied so far does not allow us to determine whether the improved results of the 
experimental group is a result of just more time on task, or whether the quality of the MASCARY 
materials contributed to the efficiency of the remedial activities. The approach of Chen (1990), in 
which he applies Mari<ov chain analysis on repeated data on educational achievement of a 
certain group of students, may shed light on this questfon. Under the assumpttons that (i) every 
student can reach a pre-defined mastery level if he spends enough time on the learning task, and 
(ii) the quality of the instruction given is stable , one can, by applying Markov chain analysis, 
calculate for each pre-defined (sub)group the expected average time to reach the pre-defined 
level of mastery. By applying this analysis to our experimental and to our control group, 
differences in expected time to reach mastery level might reveal differences between the quality 
of instruction in both conditions, and in this way help us to separate the possible two causes of 
the effect of the experimental conditions (that is more time on task, and the quality of the 
MASCART materials). 

For this purpose, the students in the experimental and control group are divWed in five ability 
categories (second method for ability grouping) based on their score on the pre-test (40 items). 
Table 5 depicts this ability distribution of the pre-test and the post-test for both the experimental 
and the control group. 



Insert Table 5 about here 



To apply Mari<ov chain analysis (MCA) on the Basic /»Igebra Test (BAT), the data on the pre-test 
and post-test for both groups need to be presented in a transitton matrix, indfcating how students 
on a certain level on the pre-test score on the post-test. Table 6 contains the transitton matrices 
for the experimental and the control group. The results of the MCA according to Chen (1990) are 
given in Table 7. 



Insert Table 6 about here 



Insert Table 7 about here 



The interpretatton is, that in the experimental group the subgrouo of students who scored in the 
pre-test on level 2, needs an average of 1 .19 TE (time units of tlie experimental group) to reach 
the defined mastery level (that is level 1 ; see Table 5). Similariy, in the control group the 
subgroup of students who are, for example, in the pre-test on level 3, needs an average of 2 27 
TC (time units of the control group) to reach the defined mastery level (level 1). As the time on 
task in the experimental group (MASCART) is greater than in the control group. Table 7 also 
contains the expected times for the experinwntal subgroups calculated in the time units of the 
control group. The data show that for the subgroups In the experimental conditfon in all cases the 
expected average time to reach the desired mastery level (level 1 , Table 5) is smaller than for the 
equivalent groups in the control conditton. 

Because of the stochastic nature of the phenomenon we are analyzing, we cannot conclude that 
the experimental conditfon is always better than the control conditfon, or that (after having 
corrected for the differences in time units between the experimental and the control group) the 



better penormance of all experimental (sub)groups is due to the quality of the MASCART 
materials. For this we have to test whether the expected average time (in both the non-corrected, 
as the corrected situation) for the experimental subgroups is shorter than for the control group. 

Chen ( i ^ ^0) also developed a procedure for testing the null hypothesis that the expected time for 
both the experimental group and the control group come from the same population; that is that 
the difference between the expected times results from stochastic observations. To test this 
hypothesis, Chen applies the Pearson chi-square procedure. In this case, the contingency tables 
are 1*4 tables: 

T(observed, experimental): 1.19 1.63 1.96 2.46 
T(observed, control): 1.75 2.273.063.21 
T(theoreticalty expected): 1.47 1.952.51 2.84 

where the both T(observed) are the expected times for the subgroups in the experimental 
condition (not conrected, see Table 7) and the control condition, and the T(theoretically expected) 
is the mean value of the other two. The Pearsor. hi-square test statistic is 13.68, while the critical 
value is 9.49 (df=4; alpha 0.05). We can reject the null hypothesis, which means that the 
expected time for reaching the mastery level is significantly shorter in the experimental group 
than iri the control group, in the case of not conrecting for differences in the unit of time. This 
result is consistent with the preceding sectton, where we tested the differences in post-test scores 
between both groups. 

Applying the test procedure on the conrected expected tinries (for the experimental group) , then 
the test statistic is 3.99 (df=4), which is not significant for alpha = 0.05. 
In 4.2.2 we found that especially the weaker students profiled from MASCART. We therefore 
applied the same test in an explorative way on only the levels 4, and 5 subgroups. The test 
statistic is 1.96 (df=1), which is not s*^nifficant (alpha =r 0.05). 

4.2.4 Conclusion 

In this section, the effects of the MASCART materials on student knowledge in Basic Algebra 
have been examined. A general improvement in basic algebra knowledge and skills could be 
measured. We could not proof that this improvement was caused by the quality of the MASCART 
materials; but the data show that the combination of increased time on task and the working with 
MASCART materials was effecth/e. 

One might argue that a cheaper way of remedial teaching might have been to give students an 
extra tutorial per week, instead of the MASCART sessions. However, when consWering the rather 
low attendance of students during regular tutorials, it is doubtful whether the same effect could 
have been reached using this kind of remedial teaching. In the PESC sKuatton, different kinds of 
remedial teaching are not relevant options, since mathematics staff does not have time to gh/e 
extra tutorials. The MASCART materials have been designed especially to satisfy this need and 
are therefore lecturer independent. So. one might say that the computer assisted MASCART 
materials have proved to be a successful means to realize remedial instruction, where at present 
no other possibilities for remedial instruction are easily possible. 

Another effect of the MASCART materials was the effect on student motp/ation. The 
questionnaire about the MASCART materials indicated that students liked to work with 
MASCART very much, because they could work indh/Wually and at their own pace. Whether this 
posith/e effect on students' motivation will last when the 'novelty' effect is gone, has yet to be 
studied in this situation. It can be concluded that indications exist that this factor for lasting 
success (effectiveness of the materials) is indeed fulfilled in the MASCART situation, although 
some improvements in the materials can be made. 



4.3 lnstituttonallMt?pn 



Because the MASCART project is not yet in its institutionalization phase, a number of factors of 
Table 1 can not yet be discussed property at this moment. However, because the implementation 
process has already taken place and because some insight in the effects of the use of the 
materials is already gained, some expectations concerning th3 future continuation of the project 
can be made. But first, the institutionalization factors will be discussed. 

4.3.1 Material level (+) 

At the material level, it was checked whether the implemented materials have indeed proved their 
effectiveness. It appeared that the students indeed gained from the MASCART sesstons. 
Therefore this factor can be considered to be fulfilled. 

4.3.2 Institutional level 

On institutional level (PESC level), there are six factors that are important for lasting success. 
Stable, routinized implementation (OU) 

From the information discussed in sectton 4.1.1 , it has become clear that the implementation took 
place in a relatively stable situation. Because the MASCART project (in combinatton with the 
Computer Awareness Course) was the first real innovatton with which the PESC Department had 
to deal, the process of implementatton can not be called routinized yet. However, no major 
problems occurred during the process of implementation. 

Inbuiltness of the project (0) 

It is too eariy to conclude that the MASCART-Basic Algebra is already "built In' in the PESC 
situation. Although there are concrete plans to continue with the Basic Algebra materials, and to 
extend the project to other areas within Mathematics and to other subjects within PESC, the 
project cannot be labelled as 'invisible' or 'widely used' yet. A positive aspect of the MASCART 
project is that, once the materials are pitoted and revised, they betong to the PESC Department 
and this departnent has full responsibility for the use of the materials. PESC Is no tonger 
dependent on external help concerning MASCART, which fosters the factor of Inbuiltness of the 
project in the department. 

Long temu policy (0) 

Just So with the above factor, it must be concluded that there is no explfcit future policy plan for 
the MASCART project at the moment. The larger PRESS project, of whfch the MASCART project 
IS a part, is planned to mn until 1991. The plans for MASCART till that time are stated in the plan 
for the total PRESS project. What is going to happen after the project period has expired is not 
known yet. 

Shared ownership and depersonalized pattems (0) 

R)r the continuatton of the MASCART project in the future, the factor of 'shared ownership' plays 
an important role. Although all staff members are eager to learn about the usefulness of the 
materials and are very interested, in practice the vtotk during the pitot phase has been done by 
the computer expert of the Mathemattes and Computer section only. This in fact leads to a 'single 
ownership', which is not an optimal conditfon for continuatton of the project. 

Interaction (+) 

Communfcatfon channels have been quite good during the piocess of development and 
implementatton of the MASCART materials. Still, emphasize must be put on the Inportance of 
communfcatton and interactton, because the literature showed that a project can only succeed 
when everyone is continually Infonned about the activities going on (Janssen Reinen 1989) 
Especially when the MASCART project is going to be incorporated in the department', Interactton 
among staff members should be planned in order to keep colleagues infonned about the activities 
of the different staff members. 
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Leadership (i-) 

As already stated earlier, the role of the Head of Doparlment is an important one during the 
process of implennentation. But his role does not stop after the phase of implementation. The 
process of continuation requires a person with a continuing positive attitude and willingness to 
support, in order to make the ultimate innovation successful. In the situation of PESC, such a 
leading figure is present and therefore, this factor can be considered to be fulfilled. 

4.3.3 Institutional transcending level 

At the institution transcending level, two factors play a role: the necessity of the change and the 
support and planning for the future of the MASCART project. For this project, the transcending 
level consists of the University of Botswana and the broader national context (the Ministry of 
Education). 

Necessity (0) 

No clear information is available about the future necessity of the project at the institution 
transcending level. Based on Botswana's sixth National Development Plan (Ministry of Finance 
and Development Planning, 1985). it can be expected that the PESC intake of students is 
continuing to grow, "^rhis means a growing number of students wtth differing backgrounds. In such 
a heterogeneous situatfon, it can be expected that MASCART and/or other computer assisted 
instruction (CAI) applicaions (see Piton, 1989) can play an important rote. Students who lack 
baste knowledge of certain topfcs can work individually on those probtem areas, without further 
assistance of a lecturer. Thus, the conclusion can be that the need for MASCART (and CAI in 
general at University level) will probably continue to be present in the future. 

It is more difffcult to say something about the future of MASCART at natfonal level. K can be 
expected that the need for computers in the education of Botswana will continue to exist, 
especially when the plans of the Ministry of Educatton to introduce five computers per senior 
secondary school are considered (see Janssen Reinen and Piton, 1989). It is not known to what 
extent MASCART fits into these plans. 

Support and planning (0) 

Again, no information is available about future support or planning by the University of Botswana 
and the Ministry of Educatton concerning MASCART. Although it is known that the University is 
very Interested in the introduction of computers in education, no definite plans about future 
support of the MASCART project or other CAI projects exist. This a'so applies for the Ministry of 
Educatton of Botswana. Although their policy is aimed at introducir g computers in educatton, 
there are no plans concerning computer assisted instrulion projf ,ts such as the MASCART 
project. 

The conclusfon can be that, although there certainly is a po ' * :tude concerning the 
computer and its application in educatton, there are no cone ,.ans at this level jvhfch point at 
a future implementatfon of MASCART in similar projects in E ana. 



4.3.4 Synthesis 

As menttoned before, for some factors whfch are important for continuatton of the project, it is too 
eariy to be fulfilled. But we also concluded, that some important implementatton factors are not 
yet fully taken into account. Especially the fact that MASCART is still depending very much on the 
efforts of one person within PESC, the computer expert of the Mathemattes sectton, might be a 
too weak basis for continuatton of the project. If MASCART is really to be 'built in' in the PESC 
Department in a depersonalised way, more people should be made responsible, so that the 
project is not tonger dependent on one person. 
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5. CONCLUSIONS AND DISCUSSION 

In this paper, factors which are considered to be important for implementation and 
institutionalization of educational changes, were investigated in a setting in a less developed 
country. It appeared that many of the factors mentioned in section 2 (Table 1), were indeed 
fulfilled in the specific setting of PESC-Botswana, leading to a rather smooth implementation of 
the MASCART materials. Concerning the process of Institutionalization, there were still some 
factors undetenDined, but no contra-indications were yet found. Eased op this case study we 
therefore conclude, that for the time being we have no reasons to doubt the validity of our 
theoretical assumptions, spelled out in section 2. We can also conclude that. If the people 
involved in the project are going to take the conclusions of this analysis into account, then it can 
be expected that MASCART will become an integral part of PESC, after the project period will 
expire. The ultimate answ iO this can only be given after an evaluation some years from now. 
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TABLES AND FIGURES 

Table 1: Factors influendny implementation and Institutionalization 
Source: Janssen Reinen, 1989 



IMPLEMENTATION 

1. Material level 

- clarity and complexity 

- consensus/conflict about the change 

- quality (and practicality) of the change 

2. Institutional level 

- central office direction, comnrltment and support 

- process for implementation and institutionalization 

- professional development and assistance (teachers) 

- professional development and assistance (others) 

- implementation monitoring and problem solving 

- principal's leadership 
-environmental stability 

- mutual support (internal) 

- user values and preferences 

- motivation and attitudes of teachers 

3. Institution transcending level 
-community support 

- environmental stability 

- national context 



INSTITUTIONALIZATION 

1. Material level 
- materials have proved their effectiveness 



2. Institutional level 

-stable, routinezcd implementation 

- inbuiltiiess of the project: - widespread use 

- invisibility 

- rise in legitimacy 

- organisational space 

- continued assistance 

- long term policy (including maintenance plan) 

- shared ownership and depersonalized patterns 
-interaction (communication) 

- leadership 

3. Institution transcending level 

- necessity of the change 

- support and planning ('managed change*) 
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Figure 1: Evaluation framework 
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experimental 
group 



01 M X 02 



control group 01 M 02 



Figure 2: Test design 

Table 2: Mean scores and standard deviations on pre- and post-test and t-test results 





Mean scores +/ 


- s,d. 


Group N 


pre-test 


post-test 


exp. 130 


26.06 +A 5.89 


32.98 +/- 4,80 * 


contr, 130 


26.11 +/-5.89 


30.96 +/- 5.44 * 


* significant difference with P< 0.05 






' ability level 

^ « pot ley: X « po6t'lorexp group) V pritlk^fcontr.group) 



Figure 3: Scores on pre- and post-test per ability group 



Tablo 3: Results of the ragression analysis for experimental group and ability 


Source 


Sum of squares 


Df mean square 


F-ratio 


P 


Group 


212.96 




16. ob 


0.00 


Ability 


1094.32 


2 547.16 


42.55 


0.00 


Group*ability 


0.84 


2 0.42 


0.03 


0.97 


error 


2623.13 


204 12.86 






Table 4: Differential 


scores (pre-test minus post*test} per ability group 








Mean differential score •«-/ stand.dev 










ability group: 








low 


middle 


high 






n=36 


n=63 


n=31 




group: 










exp. 


9.86+M.06 


6.95+/-3.07 3.42 +/ 2.42 




contr. 


n=r38 


n=61 n 


=31 






7.23 +/4.93 


4.53+/-4.07 1.38+/ 2.51 





I ERIC , 



21 



•e 




abiiity level < = p Vt' minus f» teqcontr.group) 



• = fC<*r-minus pfeVa^exp.group) 



A = differential score (exp.group) minus 
differential score (contr group) 



Figure 4: Differential scores per ability group 



Table 5: Distribution of students in the exporimental and control group 



ability level 



1 
2 
3 
4 
5 



nr. of items 
correct on BAT 



33-40 
28-32 
23-27 
19-22 
<=18 



Number of students 
exper. group control group 



pre-test post-test 



20 
43 
35 
lb 
13 



78 
38 
3 
7 
1 



pre-test post-test 



20 
41 
35 
22 
11 



57 
44 
18 
7 
3 
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Tablo 6: Pre-test/post-test traniition 



Pretest 
level 



Experimental group 



20 
43 
35 
16 
13 



Posttest level 



19 1 0 

36 7 0 

17 17 1 

5 8 1 

1 5 1 



0 
0 
0 
2 
5 



o| 

0 
0 

1 



78 38 3 7 1 
1 2 3 4 5 



Pretest 

level 1 
2 
3 

Control group 4 
5 



Posttest level 



20 


17 


3 


0 


0 


0 


41 


24 


14 


3 


0 


0 


35 


13 


15 


4 


2 


1 


22 


2 


10 


8 


2 


0 


11 


1 


2 


3 


3 


2 




57 


44 


18 


7 


3 




1 


2 


3 


4 


5 



Table 7: Expected time to reach mastery level 



Subgroup level 



2 
3 
4 
5 



Expected time * to reach level 1 
oxperimental group control group 



1.19 (1.42)" 

1.63 (1.96) 

1.96 (2.35) 

2.46 (2.95) 



1.75 
2.27 
3.06 
3.21 



* the unit for expected time is the amount of instruction between 
pre-test and post-test; for the control group 4 weeks of 4x2 lesson 
periods (Tc); for the experimental group Te is Tc plus 4 weeks of 
2 periods per week MASCART instruction; so Te = 1 .2x Tc. 

" The expected time for the experimental group is Tc units 
(multiplication factor 1.2). 
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